Jason Knight 0:00 Hello, and welcome to the show. I'm your host, Jason Knight. And on each episode of this podcast, I'll be having inspiring conversations with passionate people in and around the wonderful world of Product Management. If that sounds like the sort of party you want to get invited to, why not come and join me and some of the finest thought leaders and practitioners in the world on https://www.oneknightinproduct.com, where you can sign up to the newsletter, subscribe on your favourite podcast app or follow the podcast on social media and guarantee you never miss another episode again, make sure you tell your friends while you're at it. On tonight's episode, we welcome back a guest who's been following the ChatGPT story from the start and almost from within saw an opportunity to take a successful startup and pivot it into an AI powered product management copilot that will turbocharge your product teams. But is it going to put us all out of a job? Is it going to make us lose whatever sense of autonomy we thought we had? And are we going to hallucinate our way to product failure? For answers to all these questions and much more. Please join us on One Knight in Product. Jason Knight 1:06 So my guest tonight is Yana Welinder. Yana is a former law professor and erstwhile fairy princess children's entertainer who says she's passionate about cold plunges, although I promise you the water tonight is lovely eagerly at listeners might remember Yana from a few months back when she spoke about her mission to save the world from stupid smart things and revolutionise the usability of IoT devices with her company Kraftful. However, she has since thought better of all that and pivoted to unleash the awesome power of large language models to revolutionise product management instead, the company has been growing like mad, and she's even been a guest on Masters of Scale! But Yana still remembers her friends. And tonight, we're gonna dig into the big pivot, find out what it's like building AI powered products, and see if she's gonna feel really, really bad when she's replaced. So with robots. Hi, Yana. Welcome back. How are you? Yana Welinder 1:50 Hi, I'm doing really well. Thanks so much for having me again, really excited to chat. Jason Knight 1:54 No problem. It's good to have you back. I'm looking forward to digging into all the stuff you've been doing since the last time we spoke. But the last time we did speak craft was as discussed in the IoT business, and you had a mission to save the world from stupid smart stuff. But that's all changed. So for the record, what are you doing with Kraftful these days? Yana Welinder 2:12 Yeah, sure. So we're building a copilot for product managers in other product builders, that saves product managers, hopefully hundreds of hours by analysing user feedback from various sources like App Store reviews for tickets, user interview notes really anywhere where you have users talk about your product that could just be like Twitter mentions. And the end result is really just to have complete copilot that automates every tedious part of the job, though, we're starting in the user feedback summary part. Jason Knight 2:43 As I say, there's a lot of tedious parts of the job. But that sounds really exciting. And I can definitely imagine all the AI thread blows, right, and loads of tweets about that, and cluttering up Twitter of all their hot takes about it. But before we do talk about it in more detail, I want to talk a little bit about that decision to pivot because again, last time we were talking, you were well down that route of trying to revolutionise IoT devices. And obviously, it's all changed chat GPT has come out and shuffled quite a lot of things around. But was the pivot always on your mind? Like was this always something that you thought that you could do or should do one day? And maybe just couldn't do it yet? Or did events really just kind of overtake you and demand kind of creep up on you? Yana Welinder 3:25 Yeah, that's a great question. And I think it was that most definitely, always on my mind. In many ways, this was a very hard pivot, just in terms of what our product does and who it serves. In many ways, it was almost no pivot at all, just if you think about our mission, which was to help product managers build better products, and we're so focused on the IoT crowd, and now we're focused on a more broader product about, but most definitely, you know, something that this specific problem that we now solve is something that is a problem that I've personally thought about for a very long time. So before starting Kraftful, I was a product lead, and ultimately head of product at various tech companies. Were really gathering user feedback from various sources is always a big undertaking, particularly when we're dealing with like millions of users. And I had both B2B and B2C products in one in one company, and so had to listen to both a sales call transcripts and support tickets in App Store reviews and reading subreddits. And, you know, like, trying to follow what people were saying about our product on Twitter. And so this is this particular problem was always top of mind for me. So when I first got to play with GPT 3 back in early 2020. This is literally the very first problem. I applied it to you. And I built. So yeah, so this like, this is always on my mind. And that first prototype that I built back then, it wasn't very good. And so that was sort of like, you know, I couldn't I couldn't make that into your product back then. But what it wasn't good at was summarization. And what it was really good at was text generation because indeed that's that's the primary purpose of language models is to generate text and Not necessarily summarise. And so I kept coming back to this problem over and over and trying it again. And at some point, I discovered that actually it's getting better. And that's when we decided to create a feature in our product packs that the back to instal for for IoT for the IoT crowd. But we started summarising appstore reviews with GBT three, and had that in our product in 2021 in production, and then we continued experimenting with it in at some point that became so good. And we started seeing so much demand for it that we're sort of like, okay, actually, now we're gonna go all in on this one feature, we're gonna make that into complete product, we're going to expand it so that it's applicable to folks that aren't in the IoT space, and aren't mobile app PMs, and they can use it for anything. And that's sort of like when the full kind of hard pivot happened, which is sort of like towards the end of last year, and we launched in beta in February of this year. Jason Knight 5:56 But you could have just kept ploughing the IoT market, and just really optimising for that. So what was it that made you decide? Let's go for basically, all product managers and all product teams? I mean, aside from market size, or was it just that? Yana Welinder 6:09 I think this is just the problem that I am ultimately more passionate about to help product managers build better products. And I think that, you know, I saw a potential for doing that specifically applied to IoT when there was there weren't tools or the technology wasn't there to help everyone. And when I suddenly got to the point where I could build something really meaningful with new technology that was sort of that was just eye opening, or something. Jason Knight 6:38 That's what we all dream about. Right? But one thing with pivots is that you're generally pivoting away from something and we just talked about what you were pivoting away from, although some companies, obviously try and do kind of both, you know, try and satisfy the old market and the new market and have varying levels of success doing that, but you've gone all in on the new stuff. So I guess I do have to ask, if you had to abandon your poor suffering IoT users, are you still able to kind of cover them as well? Yana Welinder 7:04 We're still able to cover them as well, we do have users from the IoT space using our product today, they now make up a very, very tiny portion of our user base that is much, much more just general software, PMs at pretty much every tech company you can imagine. But also really cool. Other companies and organisations that are sort of retail brands, or financial institutions, or even we have cities, or the National Art Gallery, right? Like all these institutions that aren't actually tech companies at all, or really companies in the world, which I think is really cool. Jason Knight 7:38 That is really cool. But when we talked before on the podcast about the IoT crowd, we talked a lot about their passion and their opinion nativeness. Like, it's a very kind of hardcore crowd in some ways. And they have a lot of opinions, and they've got a lot of things that they want to get done and can be quite outspoken. So have you had any kind or when you started this, did you have any negative pushback, or people sitting there saying, Oh, they've abandoned us or kind of frustrated with the changes that you've made? Or did they all kind of take it on the chin and move forward and wish you well? Yana Welinder 8:10 I was really nervous about this. Actually. I was just like, the week, the week we were, we were announcing this teasers. I was just so nervous. And I mean, then I sent out that email that went out to everyone. I feel like I always just like when you're sending out an email to all users, that in and of itself, this is nerve racking. But now we I do that every week, when I announced the features, but I sent it out. And all the feedback I got back was all just super positive, people were excited to use the new products, of course, like it felt that the new product was ready for them to start using, but like back then it was an alpha. But they could literally just start and we gave them you know, we gave them discount codes. So they wouldn't feel bad about it. But yeah, everyone was super supportive, which was great to see. Jason Knight 8:56 That is good to see. I'll make sure I'll have a quick look on G2 after this just to see if that backs me up. But people might not realise. But an interesting fact about all this and the pivot and the technologies that you're using these days, you are actually married to the VP of product at open AI. So open AI who, obviously you are responsible for ChatGPT. So that does seem like a bit of a handy coincidence. But I do have to ask since your husband's on the inside. Are you already on like GPT version 5 or using some souped up version of the API? Or do you kind of just have to slum it like the rest of us? Yana Welinder 9:27 I really have to slum it like the rest of you. I think actually, I guess at this point, I get just so many requests for people who are like, Can I get you kno w, early GPT 4 access... I'm like... I am just the wife like I don't I don't get to decide these things. But I think I would say that the the handiest of coincidences really being being married to Peter is that while he wasn't really early opening I employee he basically joins in their first year and has been probably like one of the, you know, earliest employees at this point. That's the list with the company. Really, this is something he has Been really passionate about for as long as he and I have known each other, which is, you know, like 20 plus years. And so I've had a chance to just like, talk about this type of AI for like, for decades, like, I think everyone's kind of just starting to think about it. Whereas I've been thinking about how it's going to change the world for a very long time and how I would apply it to the problems I'm seeing. So of course, I would say that that's kind of like my my unfair advantage, much more so than him necessarily being an open AI employee. Jason Knight 10:31 Oh, yeah. It's no fun if you don't get some kind of special treatment, though. I'm assuming, though, that you do get to spit ball some ideas over the dinner table, which must be pretty handy as well, right? Yana Welinder 10:41 Absolutely. Yeah. I mean, he's definitely the number one advisor at this point. Well, I don't get any, like special things nowadays, particularly now that everyone's really into the stuff in the early days when like, no one wanted to use GPT. Three, which is like kind of hard to imagine that that there was there was a time, that's when he would be like, you know, this like, like during COVID. Lockdown. He's like, Can you can you try it out? Can you give me some feedback. And so I definitely definitely did get an unfair advantage there. Jason Knight 11:08 Oh, there you go. You got your first mover advantage. And now it's just time to make sure you can run the ball home or whatever. You're doing the sports in America these days. But let's talk about that copilot that you're building then. So obviously, you've talked a bit about what it does. And we definitely could talk about that in a sec. But was the technical process of pivoting and getting your team onto this new track and starting to build stuff in a different way and and integrate new technologies? Was that like a really easy process for you in the company? Or was there a bit of friction there and a kind of a bit of resetting as you tried to change path and get along a new track? Yana Welinder 11:42 I think one thing about our team that I keep being positively surprised by though I shouldn't be because I've hired these individuals. It's just how like, aligned everyone is. So the moment we talked about pivoting, I think everyone just started thinking about how they're going to make that happen. And we do quarterly offsites. And so we did this at one of our quarterly offsites. And then we just planned out the work and wake up going. But I think what really helped there was that before deciding to pivot, I did one thing, which is I went to a couple of conferences, which I like very rarely go to conferences, but around that time, I was like, actually, I'm gonna I'm gonna take some time, and I'm gonna go to conferences, and I introduced myself as the founder of this new company. And with this new mission, so like a copilots, for product managers summarising user feedback with GPT 3. So, you know, before GPT and so what I the the feedback I started getting at that point was sort of people naturally started telling me about their problems in the space. And so I got a lot of this kind of like, very early validation, that was not feedback on the product, it was kind of just like people talking about, like, ah, you know, we have our product team is like, really wants to know about all the things support knows. And so we have we created this process, but support comes in, like tell us about the same features that don't really understand that they're talking about the same feature, but they're just using different language, or, you know, support has their own agenda. They're like not telling us about the things that really matter. They're telling us about the things that will make their life easier, like these things that are sort of just like, very clear validation. And so I've put all of that together, and presented this to the team as like, Look beyond what they're doing. There is this like, huge opportunity, I know about this opportunity, because this used to be my life. But I also now just went out and gathered this additional validation so that we can kind of feel better about it, we can feel like it's not just Yana's dream, we're actually building like, many people's dream. Jason Knight 13:47 What I was gonna say, though, about the many people I mean, I'm assuming that there are a bunch of people out there now trying to hop on that bandwagon as well. Now, obviously, to an extent you had at least early understanding of where this might be going. So maybe that gave you a bit of a first mover advantage there. But I'm sure that every single product management tool out there now is trying to cram something like this into it, or into that tool. And given that it's the same underlying technology beneath it, like the Chat GPT4 now that I presume it's using. How do you differentiate that offering? I mean, obviously, you could build stuff on top of it, but like how do you specifically differentiate from all the other tools that can use the same underpinning technology? Yana Welinder 14:27 I think there's a few different ways in which we differentiate from different tools depending on if they're like the incumbent product tools versus the the new tools with the incumbent product tools, I think, I think it is something that actually one of our one of our customers put much better than I could, which is that Kraftful is Gen AI first, or like forward actually, I think was the term he used Gen AI forward kind of like, you know, either designed forward or actually what I thought was comical, it's more like, like fruit forward wine, you know? If Right. But this idea that we're really using LLM in the best possible way to use them, rather than trying to plug them into an existing tool and try to make that tool slightly better. And that I think, is really what makes our solution so powerful compared to some of the kinds of folks trying to plug LLM into their... Jason Knight 15:22 JIRA, into JIRA. And that's got to be in JIRA. At some point. Yana Welinder 15:26 There is definitely an LLM in JIRA already. And then, in most most of these other tools, but I do think that, you know, we're not trying to replace JIRA, in fact, we're now building an integration to make it easy to write JIRA tickets with the user feedback that is incredible. And so we don't think ourselves to secure a replacement. But But I do think that there's something about being able to live between all of these different things and then build with the technology in that technology native way, as opposed to just like trying to make it work for for your existing products. Jason Knight 16:00 No, absolutely. In fact, the integrations point is an interesting one, which I was about to come to. So let's talk about that, like you're talking about something to summarise, as you put it, App Store reviews, support tickets, user interview notes, sales calls, transcripts, survey results, presumably any other kind of textual data as well. But there are a lot of tools out there that do all of those things. And you've already talked about integrating with them, but there are a lot to integrate with. So how are you approaching your, I guess, integration roadmap where you can sit there and say, Okay, well, you know, maybe something like JIRA is obvious because everyone uses it. But is it simply just reverse sorting stuff by how many people use it? Or if you've got some other strategy to work out? What you can most meaningfully integrate with? Yana Welinder 16:40 In the early days. And it's funny to talk about early days since we launched the product literally three months ago and beta last week. In the early days, we looked at two things, which is how many people request that and what is the source in that? Like, how well can we summarise it? Right? Because there's big differences between these different sorts of trades, like user interview notes, super dense with product insights, sales call transcripts, not so much. So there's a lot of complexity. Jason Knight 17:07 Oooh, don't be so harsh on the salespeople. Yana Welinder 17:10 Well, the salespeople are talking a lot, right? So you're not getting a whole lot of what the other person is saying in the transcripts. So those are like basically what we looked at those two, right? Like what people were telling us, give me this source, and then we were looking at, like, what can we do well, and nowadays, we now just have like this super long list of like, I think 100, like hundreds of user requests for different types, for different sources, hundreds of sources, I should say, and 1000s of requests supporting them. And so what we now do is we've essentially created a way to up votes, the most common ones in our product, so that when people sign up if they're, if there's something that doesn't make sense, but they can essentially tell us actually, this is something I want to use. And then we have another box for like requesting your own thing that's not there. And that just gives us a sense of in addition to like, what's most frequently used, and what can we do well, what are our users looking to do, right? Because we really want to prioritise those folks, and not just generally like the market more broadly. Jason Knight 18:10 Makes a lot of sense. But that does beg the question, are you using Kraftful to help you design Kratful, and eating your own dog food entirely and emptying out the can into your bowl? Are you all in using Kraftful to build your own application? Yana Welinder 18:24 We are most definitely using Kraftful for Kraftful. And that just had to be true, because we're like a tiny company. And, you know, we haven't we haven't talked about our user numbers. But I would say that I haven't in my career, and I've worked I've been a pm at early stage unicorns in that, you know, I've led products with millions of users. I haven't seen this kind of growth before. And so we've had pretty significant growth, lots and lots of user feedback. And we're just so few people that are able to actually mean this. So we had to, we had to very quickly integrate raffle into the process to make it work. Jason Knight 19:00 Well, you don't get on Masters of Scale for free, I guess not like this podcast. But as an example, then, what's one piece of actionable feedback that you've acted upon from all of those different sources of feedback that you got, and all of the different stuff that was coming in, like one way that craft or helped you craft or kind of sort through that and surface and insight that you then acted on? Yana Welinder 19:23 Lots and lots of things... I'm trying to intend to think of something, something concrete, but basically, you know, we launch, I send out this weekly email to users that has multiple features in each email. And each of those are based off of feedback that we've gotten from users to give you just a concrete I think, in the early days, we really get them the earliest... Jason Knight 19:44 Two weeks ago! Jason Knight 19:45 Two weeks ago. We thought that it this would really, really just be a tool you'd use for basically like sprint planning, or you know, kind of like just in time. So you was really clear about your most recent feedback and what people came back to us and said I Actually, I do care about the older feedback, I want to, I want to summarise my older feedback to us, we had to build that into our product very quickly, which meant that, you know, that just means we have to be able to take a huge load of user feedback and summarise it in multiple stages. Another example that we're about to launch very shortly, that is that it's all been based off of user feedback we send out, you know, we pull in this data periodically, which is every day, and then we send our users a weekly summary of their user feedback via email. But what we've heard a lot is I don't want to get this via email, I want to connect this to Slack channel, and have all of my team see all of these summaries. And so we're very close to launching a Slack integration to be able to do that, Jason Knight 20:39 Oh, there you go, all getting connected. And we'll never be able to let people have any time off at all. But if we then concentrate on some of the more positive aspects of what you're doing, and the tool that you're building, and the use cases that it opens up, and the things that maybe you can do now that you couldn't do before. Let's imagine we fast forward, you say you're scaling pretty well, presumably going to become a de facto solution for AI powered Product Management at some point. But how do you think that this will change Product Management for the better? Yana Welinder 21:10 I think that there's so many things that product managers do in their day to day, that isn't really, that is really important, because it has to happen, but isn't really critical to what a great product manager does a really what a great product manager wants to do. And so a big piece of that is summarising user feedback. So, okay, check, we've got that out of the way. But the hopefully, but really, I think, at a point, just to give you an example, right, like when if you if we can summarise all of the user feedback. And we can also summarise all of the internal discussion that led to a product or feature. And you have all of that summary. And you can imagine, we then help the product manager, articulate the product requirements around that. So then the engineer on the other end is getting this ticket that is well articulated always has the why, and also has a link. And you know, I think about an engineer at that point, the engineers probably not a code writer, either, they're probably more of like an AI operator. But that's let's put that aside for a second. But the, the engineer will then have access to like to be able to click through and see a summary of the user feedback in the team conversation that leads to the feature they're working on. So they can see obviously see the why and a much clearer way, but also be able to get at kind of the details, and then be able to go back and say actually, the details aren't as well defined, I want to ask the users a follow up question and then be able to do that really easily. So you can kind of imagine it's removing a lot of the back and forth on the team so that everyone can focus much more on the strategic work. And the kind of the bigger, like, how do we how do we delight users in a really supercharged way? Or like a really leveraged way? Jason Knight 22:59 Do you think that we're going to maybe be replacing people doing user interviews by sending out a chatbot powered by a ChatGPT version, whatever, to go and just get all of the information in the first place? So do you think that humans are going to be always needed to do that sort of stuff? Yana Welinder 23:13 I think it absolutely will, we will replace that out of replace humans. And I think what really what I'm really excited about this that for in the context of user interviews or surveys, you're gathering a small group of people, and you're asking them a tonne of questions. And by the end of that questioning, usually the user has no recollection of what it's like to use your product, right. But it's just so so removed from from that, from that context, what I think is really exciting is to be able to replace that whole information gathering with something that goes out to so many more users. So you can get a much bigger cross section of your user base. And ask them one question that's related to feedback that they've previously provided and as much more contextual, and it's, it's much easier for them to answer it. And then once they do, they're then kept in the loop on how that feedback actually impacted the product. Like there's so many ways we can help product managers build a community of happy users around their product with this kind of discovery process. That is just not possible today. But we have small group of people answering lots and lots of questions. Jason Knight 24:19 That's if we have any people answering any questions in some companies, of course, but that's a whole different podcast. But as some people have been saying the chat GPT is coming for all of our jobs, and that knowledge workers are going to be massively affected. Loads of people going to get laid off. And you know, who knows what's going to happen after that? So do you think that tools like craughwell will put at least some product managers out of work? Yana Welinder 24:42 I think that net effects will be that there will be more work for product managers. And that's not specifically because of craft but because of AI. Generally, what what I mean by that is that it will be much easier to start a company when you don't need as much capital to start a company. So we will be able to go out and start solving all of these things that we see is considered to be like not venture backup all business, but it's actually venture scale businesses. So consult like all of those niche cases, and all of those companies will need product managers, because product managers are kind of the strategy behind the product you're building. And you will need humans, I think like for as long as we need humans to do any professions, you will need product managers to do product management. That's just, that's just what it is. But I do think that, you know, AI enabled product managers will then do an effective job at that, but they're not going to be replaced. They're just going to be supercharged. Jason Knight 25:42 But is there a danger of product managers developing kind of a dependency on this AI tooling. So we're going to start using tools like craft, or we're going to be accelerated beyond our wildest dreams in ways that we never thought possible. And, you know, I can accept that there's gonna be some really exciting outcomes from that. But at the same time, maybe not the current crop of product managers, because we already know how to do some of this stuff, but maybe like the next generation of product managers, or some of these people that have enabled to go and do something that they didn't know that they could do before through the power of technology, but they kind of lose the ability to actually think or do some of the stuff that they're now automating themselves. And then if their internet connection goes down, they're stuffed, like, do you feel that we're kind of whilst empowering, PMS were also to some extent, disempowering PMs. Yana Welinder 26:35 I think a good analogy to this is the different map apps, right? Like there used to be a time when we were all had to figure out how to find things in a city. And then all these tools came around, and now we just rely on them to get to places and we just in so we like that frees up our mind to be able to do other things with that, that space of our brain. But you know, when our phone is out of battery, we certainly don't don't know how to get places. It's mostly not a big deal, like... Jason Knight 27:10 But I can still get like a paper Atlas, if I really need to buy a road atlas or something like that. The short, I'm not going to have an atlas that has road information for the entire world, in my car. But I'm going to probably have one that at least can get me to the next city or something if I'm really desperate, or maybe I just know, because there are signposts. Is there an equivalent for some of the stuff that you're enabling? Because, you know, you can't just print out Chat GPT. It's just, there's, there's a certain dependency on that being available for you to do this stuff. But is there any kind of way that you can kind of backfill for the times when it's not available? Or maybe when you just get a bit bored of looking at it, and you just want to try and attempt things a different way? Like is there any backup plan? Yana Welinder 27:55 I think all the like the product material that's currently out there? Is that right? Like it is that old Atlas, that you could, you could go back and read blog posts about how to write a good PRD. Right, we could... Jason Knight 28:09 There's a few of those posts around. Yana Welinder 28:11 That's right. Right. So I think I think we have analogue. I don't know I'm calling that analogue... like a blog post is now analogue. That's where we come... Jason Knight 28:21 Well, you know, it's been at least three weeks since we started, right. Yana Welinder 28:23 That's right. That's right. Yeah. So I mean, I think I think that there will, there will most definitely be materials that folks can use when they're not using these tools, I don't think they will be using them much to be honest. They will just wait for that internet connection to come back. Jason Knight 28:38 Because all their products will start working as well anyway. So I guess it's all part of the same system. But speaking about dependencies, is it not somewhat concerning to be effectively giving all of our data, I mean, not all of our data, obviously, but a lot of our customer feedback data, potentially, we have some sensitive or personal information in there, depending on the source of the information and what they said in the interviews and what metadata we keep about them. We're dumping that all into a big AI bucket, which is then going to do all that cool stuff to extract meaning from it and give it back to us. But at the same time, there are companies out there that are pretty risk averse, and they might not want to do any of that stuff. Is there a way to mitigate those concerns so that for example, a company that is a little bit more risk averse, can maybe use graph or get all the benefits out of it without worrying too much about either GDPR violations or other privacy problems that maybe just make them just not want to even take the chance? Yana Welinder 29:35 Yeah, absolutely. We do have at this point, all of the data that we send right now to open AI for analysis is not being used to train the model for example, it is being automatically deleted within 30 days. And then we are looking at options to be able to instantly delete it for certain enterprise customers. So there's most definitely as some thing that that we're looking into and constantly thinking about, like, what's the is there ways we can do this in a way that will satisfy kind of the the more risk averse companies? Jason Knight 30:09 Yeah, I'm thinking about banks and people like that, or government organisations that might be a little bit averse to even the most vanilla technology out there that they could potentially use that learn something as groundbreaking, but also kind of not understandable as much as maybe some of the other stuff that they're using. But it's good that you've at least got it on your mind. Yana Welinder 30:27 We absolutely had to have this on their minds, because we had this really interesting day, move outside the early days. Right? Jason Knight 30:36 Four weeks ago, four weeks ago. Yana Welinder 30:38 So right after we launched, we had this one day when everyone in Brazil signed up for Kraftful, which is this, like... Jason Knight 30:44 That's a big country as well! Yana Welinder 30:48 It was a big country. And we were just being swamped by signups from Brazil. And a lot of them came from financial institutions. And there's like this one woman when V bank of Brazil signed up for like someone at the Bank of Brazil that like Bank of Brazil, and we were like, What is going on like? And so it's what we discovered was that there was this UX researcher at a financial institution in Brazil that that recorded a demo video of pratfall in Portuguese. Wait, we have no idea what he was saying, we found this like Instagram video, we're like, what is going on? Hopefully, you're saying good things. And most obviously, he will say good things, because they were all signing up. But we got presented with a lot of the kinds of questions that financial institutions have very, very early on because of that. Jason Knight 31:30 Well, again, it's always good to kind of get ahead of the curve before everyone else starts asking those questions as well. But there's another thing that keeps coming up when we talk about things like Chat GPT. And that's the hallucination issue. So for the benefit of the listeners out there to to listeners that haven't been reading about this, this idea that basically chat GPT is really, really good, or can be really, really good at writing really credible looking text, that is complete nonsense. Now, there have been some lawyers in the news recently. And I know that your former law professor, so maybe you've been following this story, they've been getting into trouble for basically citing fake cases, in their arguments. And kind of getting hauled over the coals for that, and obviously, rightly so. So I'm assuming that that kind of story resonates with you. But do we run the risk of committing similar kinds of hallucination errors, if we're putting all of our stuff into ChatGPT or into Kraftful, and then using ChatGPT, and ended up coming up with some complete nonsense that we actually shouldn't be doing at all. But probably because we've also developing a dependency on it, that we're not necessarily checking that that's actually the thing that we should do. Yana Welinder 32:37 Yeah, so I think there's there's a few different things about hallucinations. The first there is that AI years are really, really like dog years. Jason Knight 32:47 Well, the other way around, Yana Welinder 32:49 Oh, and this, this technology is evolving really, really quickly. So I don't expect us to see much hallucinations in LLM very shortly. And I say this, you know, having, having experimented with with this technology in like early 2020. It was like almost all hallucination, it was just like so much legislation going on. And so just having seen it evolve over the past three years in the evolution has really expedited towards the end, I don't expect this to be a problem shortly. But while it is a problem, I think there's a few different things about it. One is that it becomes more of a problem when you use LLM, it's like a Swiss army knife. Compared to when you when you like specialise as a ChatGPT, I think opposite of like the Swiss Army knife. And when you use it in a kind of a more specialised way, it's kind of like a craft tool. There's, there's definitely other tools out there that are very specialised on a specific use case. And when you do that, the context becomes so much more clear, that makes it much less likely to hallucinate. And the reason is, sort of, I think a lot of hallucinations are a data problem. So the prompts that you use to get get the output really determines the the whether it's likely to listening is sort of like, you know, garbage in garbage out kind of problem. And, in the prompt contains two things, particularly in our case, right? We have the contexts in case the user feedback, and the instructions, right? Like when we tell GPT that, you know, take this user feedback and summarise it in this particular way to get these kinds of products, insights out of them. And the better those two things are, the less likely you are to get hallucinations. And kind of to make that even more concrete. If we have bad data that contexts where we have very, for example, we have very little insights in the user feedback. And then we're telling it give me five feature requests based on this data. But there isn't five feature requests in that data. So then, of course, it's gonna get but it's helpful. So it will give you five feature requests that did not exist, right. Yep. But that's a data problem. And so I think there's there's a lot of things you can do We've done a tonne of work to mitigate hallucinations in our output. And it's gotten to the point where it's very, very minimal. But the other thing that I think you can do is to build in controls in your product to make it really easy for the user to cross check the output. Yep. So we, we have lists where you look at the summary. And you can see how frequently that each come up each feature requests come up. And then you can click on that, to see those mentions in contexts. The biggest reason you do that is to just learn more about what users are saying, right. But it also helps you double check that it's working. It's doing what it's supposed to do. Jason Knight 35:39 Oh, still playing a tasks for those humans yet. But let's end on a positive note. I'm sure you've got some pretty cool plans coming up and new features that you're building, hopefully, based on that feedback that you're processing through and talk. Have you got any big plans? What's coming up over the next few weeks? I guess it's moving so quickly, like stuff that's coming up that you can share with us? Yana Welinder 35:59 Yeah, I mean, I think we have a lot of different ways, like new ways we're going to be analysing the data, the stuff I'm actually most excited about is just how we're improving the analysis. So summaries will just get much, much better. But there's a lot of other kinds of tools that are kind of like what I alluded to previous like sharing the summaries in Slack, being able to export it to JIRA, and automatically write the JIRA tickets with user stories and acceptance criteria and all that stuff. Lots, lots and lots of different ways to analyse and use the insights. But, you know, this is a fast evolving space. So stay tuned. Jason Knight 36:32 Yeah, well, we'll have to keep our eyes open and maybe use some kind of chat GPT enabled thing to keep on top of your release notes as well. But where can people find you then after this? If they do want to find out more keep track on what's going on at crawl forward, chat about AI in general, or maybe get you to help them sort out all those fake legal cases, they keep putting in their documents? Yana Welinder 36:52 Please don't reach out to me for legal cases. Jason Knight 36:54 But come on, you've got the expertise. Right. Your the last line of defence? Yana Welinder 36:59 Sure, sure. But still, please don't. I haven't I haven't been a lawyer for a very long time at this point. But do you but do you reach out to me generally, I am @yanatweets on Twitter, that's probably where I'm most active, or reach out to the Kraftful team with just just @kraftful on Twitter as well. And I mean, there's also ways to you know, if you sign up for Kraftful, you can contact us via that Intercom bubble at the bottom of the page. Jason Knight 37:25 I'm sure there are no chat bots involved anywhere in the chain, and it just gets straight through to a real human. Well, I'll make sure to link it all into the show notes anyway, and hopefully I'll get a few people taking a plunge in your direction. Well as ever, ya know, it's been a pleasure, obviously, wish you all continued success with Kraftful. Maybe I'll even sign up and check it out myself. Obviously, we'll keep chatting. But as for now. Thanks for taking the time. Yana Welinder 37:46 Thanks so much for having me. This a lot of fun. I really enjoyed it. Jason Knight 37:51 As always, thanks for listening. I hope you found the episode inspiring and insightful. If you did again, I can only encourage you to hop over to https://www.oneknightinproduct.com, check out some of my other fantastic guests, sign up to the mailing list or subscribe on your favourite podcast app and make sure you share your friends so you and they can never miss another episode again. I'll be back soon with another inspiring guest but as for now, thanks and good night.